Selective sync #834

okdistribute · 2017-07-29T18:42:43Z

Adds selective sync. This is a workflow requested by the Sloan Digital Sky Survey researchers at Berkeley Lawrence Lab.

New features added:

dat clone <key> --empty : will create a sparse archive and won't download any files by default.
dat sync <dir> --select=<remote-file>,<remote-file>,<remote-file>: will download remote files with the selected names
OR
dat sync <dir> --select-from-file=<local-file-list.txt> will parse the file and download all files listed (separated by newlines). Default is .datdownload

For example, check out this short video for a demo: https://www.dropbox.com/s/ih0pu1wqt2c1h9b/selective-sync-demo.mov?dl=0

Things this PR doesn't do yet:

Exit when done downloading the selected files.
UI for progress

joehand · 2017-08-02T00:26:17Z

Sweet!

This looks good. The only concern I have is that this is stateless. If you run:

dat sync --select=my-file.txt

Every time you want to update my-file then you have to make sure to include it again. If you run dat sync accidentally or without specifying the file:

dat sync

Then it'll download everything.

In my mind this was a blocker before with the selective sync. Based on your conversation with SDSS folks, do you think we should be worried about this?

We could make it stateful by adding a .datdownload or similar file to track what files to keep in sync.

ralphtheninja · 2017-08-02T01:34:35Z

@Karissa What if you want to sync a remote file named file.txt compared to only syncing file names in a local file called file.txt? How would you distinguish between the two? I know it's a silly example :)

ralphtheninja · 2017-08-02T01:39:25Z

src/parse-files.js

+
+  try {
+    if (fs.statSync(input).isFile()) {
+      parsed = fs.readFileSync(input).toString().trim().split('\n')


Could this mean problems for files on windows ending with \r\n?

Changing it to .split(/\r?\n/) would prob work

good point!

ralphtheninja · 2017-08-02T01:50:04Z

I would split the functionality into two different arguments, e.g. --select and --select-from-file. This also solves the problem of ambiguity.

mafintosh · 2017-08-02T18:53:19Z

Want this feature so bad :) I think we should use Beaker style filters dat://{key}/{folder}/{file}

mafintosh · 2017-08-02T18:55:06Z

src/lib/selective-sync.js

+    debug('archive update')
+    bus.emit('render')
+  })
+}


I'm gonna steal most of this and put in hyperdrive later

mafintosh · 2017-08-02T18:55:30Z

In general this looks solid 👍 👍 👍

okdistribute · 2017-08-02T20:37:10Z

@ralphtheninja yeah i think you bring up a good point, but I agree with @joehand that perhaps having some state file (like .datdownload) would make even more sense, so that later dat sync commands would respect the same parameters.

okdistribute · 2017-08-02T21:12:50Z

OK I made some changes so now:

select-from-file allows you to provide a file. It defaults to '.datdownload' so that multiple dat sync commands will follow the same parameters.
select now only accepts a comma-separated list of files to sync.
Windows-style newlines will work.

TODO in a different PR:

Support dat://<key>/dir/file.txt links

joehand

.datdownload file stuff looks good. Last questions:

Should we read/check that file for the regular dat sync and dat pull commands, so we avoid accidental downloads? EDIT: misread the code and this works for dat sync already ✨
Do we want to write to .datdownload when you pass --select=file.txt,etc.?

joehand · 2017-08-03T17:25:48Z

package.json

@@ -1,6 +1,6 @@
 {
  "name": "dat",
-  "version": "13.7.0",
+  "version": "13.8.0",


I'm gonna try to get #828 released for 13.8 too, can we increment version separate from the PR?

joehand · 2017-08-03T17:27:50Z

src/commands/clone.js

@@ -8,6 +8,11 @@ module.exports = {
  ].join('\n'),
  options: [
    {
+      name: 'empty',
+      default: false,


Add boolean setting.

joehand · 2017-08-03T17:29:06Z

src/lib/selective-sync.js

+    archive.readdir(dirname, function (err, entries) {
+      if (err) return bus.emit('exit:error', err)
+      entries.forEach(function (entry) {
+        emitter.emit('download', path.join(dirname, entry))


Whats the benefit here of using EventEmitter instead of calling download(entry)?

The idea was we could also figure out when it was done and call emitter.emit('end') .. right now it just stays open and you have to watch the download speed

joehand · 2017-08-03T17:30:10Z

src/lib/selective-sync.js

+  function download (entry) {
+    debug('selected', entry)
+    archive.stat(entry, function (err, stat) {
+      if (err) return bus.emit('exit:error', err)


I'd be nice if we can warn without exiting if they select a file that doesn't exist. But that may be a harder UI problem we can solve later.

joehand · 2017-08-03T18:07:14Z

src/commands/sync.js

+      name: 'selectFromFile',
+      boolean: false,
+      default: false,
+      help: 'Sync only the list of selected files or directories in the given file. (default: .datdownload)',


How do I use the default? I thought dat sync --selectFromFile would use .datdownload by default.

Then tried this:

❯ dat sync --select /Users/joe/code/node_modules/dat/src/commands/sync.js:80 if (opts.select) opts.selectedFiles = opts.select.split(',') ^ TypeError: opts.select.split is not a function

you just use dat sync and it'll use .datdownload by default.

lol @ me trying to be too fancy 🤦‍♂️

joehand · 2017-08-03T18:10:26Z

It'd be cool to put a message after dat clone --empty on how to use it:

❯ dat clone datproject.org/rmg/librariesio-csv . --empty          
Dat successfully created in empty mode. Download files using pull or sync.

joehand · 2017-08-03T18:15:48Z

src/lib/archive.js

@@ -14,6 +15,7 @@ module.exports = function (state, bus) {
    if (state.opts.http) serve(state, bus)

    if (state.writable && state.opts.import) doImport(state, bus)
+    else if (state.opts.sparse) selectiveSync(state, bus)
    else download(state, bus)


It'd be cool to have a download progress UI =). I think we can update the dat-node/stats to work for this but that can wait.

yeah, maybe lets open a PR on that

okdistribute · 2017-08-03T18:26:19Z

@joehand It already gives that message on clone. Or what kind of message were you thinking?

joehand · 2017-08-03T18:33:32Z

@joehand It already gives that message on clone. Or what kind of message were you thinking?

Ya just some more instruction on how to use the selective stuff:

❯ dat clone datproject.org/rmg/librariesio-csv . --empty          
Dat successfully created in empty mode. Select which files to download with dat sync:

dat sync --select=readme.md,my-data.csv

Or create a /.datdownload file with your file list:

readme.md
my-data.csv

And run `dat sync`.

Part of the issue was that dat pull doesn't work with selective so that message is a bit misleading right now.

joehand · 2017-08-03T18:50:20Z

src/commands/sync.js

@@ -21,6 +21,19 @@ module.exports = {
      abbr: 'ignore-hidden'
    },
    {
+      name: 'selectFromFile',
+      boolean: false,
+      default: false,


Can we set the default here? The --help prints out the default, so this was why I got confused:

dat sync --help ... --selectFromFile, -select-from-file Sync only the list of selected files or directories in the given file. (default: .datdownload) (default: false) ...

oh i didn't realize that. yeah sure

okdistribute · 2017-08-03T20:32:12Z

Now the UI looks like this when trying to download something that doesn't exist:

joehand

✨

freedba · 2019-03-05T15:07:04Z

selective UI for progress ,when complete ?
Thank you

gtignore: tests/ -> test/

6d46485

okdistribute force-pushed the selective-sync branch from ee25724 to 2c6d311 Compare July 29, 2017 18:52

okdistribute requested a review from joehand July 29, 2017 18:57

ralphtheninja reviewed Aug 2, 2017

View reviewed changes

mafintosh reviewed Aug 2, 2017

View reviewed changes

joehand reviewed Aug 3, 2017

View reviewed changes

Add selective sync API

ec7037e

okdistribute force-pushed the selective-sync branch from 4403259 to ec7037e Compare August 3, 2017 20:53

joehand approved these changes Aug 3, 2017

View reviewed changes

okdistribute merged commit 9a7150a into master Aug 4, 2017

okdistribute deleted the selective-sync branch August 4, 2017 00:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Selective sync #834

Selective sync #834

okdistribute commented Jul 29, 2017 •

edited

Loading

joehand commented Aug 2, 2017 •

edited

Loading

ralphtheninja commented Aug 2, 2017 •

edited

Loading

ralphtheninja Aug 2, 2017

mafintosh Aug 2, 2017

okdistribute Aug 2, 2017

ralphtheninja commented Aug 2, 2017 •

edited

Loading

mafintosh commented Aug 2, 2017

mafintosh Aug 2, 2017

okdistribute Aug 2, 2017

mafintosh commented Aug 2, 2017

okdistribute commented Aug 2, 2017 •

edited

Loading

okdistribute commented Aug 2, 2017 •

edited

Loading

joehand left a comment •

edited

Loading

joehand Aug 3, 2017

okdistribute Aug 3, 2017

joehand Aug 3, 2017

okdistribute Aug 3, 2017

joehand Aug 3, 2017

okdistribute Aug 3, 2017

joehand Aug 3, 2017

joehand Aug 3, 2017

joehand Aug 3, 2017

okdistribute Aug 3, 2017

joehand Aug 3, 2017

joehand commented Aug 3, 2017

joehand Aug 3, 2017

okdistribute Aug 3, 2017

okdistribute commented Aug 3, 2017

joehand commented Aug 3, 2017

joehand Aug 3, 2017 •

edited

Loading

okdistribute Aug 3, 2017

okdistribute commented Aug 3, 2017

joehand left a comment

freedba commented Mar 5, 2019

Selective sync #834

Selective sync #834

Conversation

okdistribute commented Jul 29, 2017 • edited Loading

joehand commented Aug 2, 2017 • edited Loading

ralphtheninja commented Aug 2, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ralphtheninja commented Aug 2, 2017 • edited Loading

mafintosh commented Aug 2, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mafintosh commented Aug 2, 2017

okdistribute commented Aug 2, 2017 • edited Loading

okdistribute commented Aug 2, 2017 • edited Loading

joehand left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joehand commented Aug 3, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

okdistribute commented Aug 3, 2017

joehand commented Aug 3, 2017

joehand Aug 3, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

okdistribute commented Aug 3, 2017

joehand left a comment

Choose a reason for hiding this comment

freedba commented Mar 5, 2019

okdistribute commented Jul 29, 2017 •

edited

Loading

joehand commented Aug 2, 2017 •

edited

Loading

ralphtheninja commented Aug 2, 2017 •

edited

Loading

ralphtheninja commented Aug 2, 2017 •

edited

Loading

okdistribute commented Aug 2, 2017 •

edited

Loading

okdistribute commented Aug 2, 2017 •

edited

Loading

joehand left a comment •

edited

Loading

joehand Aug 3, 2017 •

edited

Loading